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CROSS-REFERENCE TO RELATED APPLICATION 

[0001] This is a continuation of U.S. Patent Application No. 09/087,253, filed May 29, 
1998. 

BACKGROUND OF THE INVENTION 
1 . Field of the Invention 

[0002] The present invention generally relates to digital cameras. More particularly, 
improved digital camera architectures and components are described. 



2. Description of the Related Art 

[0003] Recently, digital cameras have become very popular. The digital camera converts 
an optical image to electronic image data and digitally records the image data on a storage 
medium. When the image is reproduced, the recorded digital data is retrieved from the storage 
medium and displayed on a display device or printed out as a hard copy image. 

[0004] Referring initially to Figure 1, conventional digital camera systems 100 typically 
include a digital camera 102 having an optical lens 104 for focusing light rays onto an imaging 
capture unit 106. The imaging capture unit 106 typically relies upon an array of light sensitive 
photo optic cells 108, capable of converting the light rays received from the optical lens 104 
into representative analog signals. Most commonly, the photo optic cells take the form of 
charge-coupled devices (CCDs), although other devices such as CMOS receptors may be used 
as well. As is well known in the art, each CCD array 108 must have associated with it a 
specific color filter array (CFA). In most applications, the CFA is an empirically derived 
pattern of individual color filters each associated with a specific CCD cell in the CCD array. A 
color converter circuit 110 then uses a particular interpolation algorithm associated with the 
specific CFA to generate the analog signals representing the CCD manufacturer's 
predetermined concept of the proper color scheme of the image. Manual input controls 103 
(i.e., push-buttons, for example) provide manual inputs to a step controller 105 suitably 
disposed to provide control signals to the imaging capture unit 106. Such control signals are 
used to control such imaging parameters as f-stop, exposure, zoom, focus, and flash 
attachments, if appropriate. 

[0005] The imaging capture unit 106 sends the analog signals representing the image first to 
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an analog-to-digital (A/D) converter unit 112. The A/D converter unit 112 converts the 
representative analog signals into digital signals representative of the image. The digital signals 
are then passed to a digital signal processor (DSP) 1 14 where they are converted to appropriate 
digital formats. An image compression circuit 116 as well as a memory 118 both receive the 
appropriately formatted digital signals. The image compression circuit 116 operates to digitally 
compress the received digital images in order to reduce the amount of resources required to 
further process the digital signals. One such formatting scheme referred to as JPEG is 
commonly used, although there are a wide variety of suitable picture formats. Once the image 
has been digitally compressed, it is sent by way of a memory interface 120 to a memory slot 
122 capable of receiving a memory card 124 suitable for storing the compressed digital signals. 
Such memory cards 124 include "floppy" disks, flash EPROM cards, R/W compact disc (CD), 
SmartMedia and the like. 

[0006] Unfortunately, conventional digital cameras have several important limitations. One 
such limitation is the fact that the conventional digital camera may only use the color filter 
array (CFA) and its associated color interpolation algorithms. Any subsequent improvements in 
CCD array technology cannot easily be incorporated into the conventional digital camera 
system. 

[0007] Lack of integration also provides for expensive and cumbersome digital cameras. 
As can be readily seen, the conventional digital camera system 100 has many discrete 
functional blocks encompassing many different circuits. It is therefore relatively expensive for 
conventional digital cameras to provide image correction, color correction, image compression, 
as well as provide all appropriate control and timing signals in one unit. 

[0008] In addition, each CCD or CMOS imager has different clocking requirements, 
resolution, etc. That requires dedicated ASICs, or other dedicated logic, to properly drive them. 
This results in higher manufacturing cost and an inflexible camera architecture. 

[0009] It would be advantageous and therefore desirable to have available components that 
can be used by a variety of digital camera manufacturers, regardless of their specific image 
sensor, color interpolation scheme, etc. 



SUMMARY OF THE INVENTION 
[0010] A digital image processor for use in a digital camera and methods of using a digital 
camera are disclosed. In one aspect of the invention, a digital image processor for use in a 
digital camera is disclosed. The digital camera has an image capture unit arranged to output 
analog signals that represent a captured image, a converter unit for converting analog 
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representations of the captured image to digital representations of the image, a memory for 
storing digital representations of the image, and a processing unit capable of directing 
components included in the digital camera. The digital image processor includes an input data 
stream processor for preprocessing images received from the converter unit and storing the 
preprocessed images in the memory. The digital image processor also includes an output data 
stream processor arranged to receive preprocessed images and postprocess the retrieved images 
into a viewable form. 

[001 1] In another aspect of the invention, a method of forming a viewable representation of 
an image using a digital camera is disclosed. The method includes the following operations. 
First, an image is received at the image capture unit to form a captured image followed by 
outputting analog signals representative of the captured image from the image capture unit. 
Next, the analog signals are converted to digital image signals representative of the captured 
image. Next, the digital image signals are preprocessed to form a digital image, such that the 
preprocessing includes correcting non-uniformities in the captured image. Next, the 
preprocessed digital image is stored in a generally accessible memory that is part of the digital 
camera. Finally, the preprocessed digital image is retrieved from the memory and 
postprocessed into a viewable form. 

[0012] Other aspects and advantages of the invention will become apparent from the 

following detailed description, taken in conjunction with the accompanying drawings, 
illustrating by way of example the principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0013] The present invention is illustrated by way of example, and not by way of limitation, 
in the figures of the accompanying drawings where like reference numerals refer to analogous 
or similar elements to facilitate ease of understanding and which: 

[0014] Figure 1 is a block diagram of a conventional digital camera system; 

[0015] Figure 2A is a block diagram of a digital camera system in accordance with one 
embodiment of the invention; 

[0016] Figure 2B is a block diagram of an implementation of the digital image processor 
shown in Figure 2A; 

[0017] Figure 3 is a block diagram of a digital camera system in accordance with another 
embodiment of the invention; 

[0018] Figure 4 is a block diagram of a digital image processor formed in accordance with 
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an embodiment of the invention; 

[0019] Figure 5 is a functional block diagram of a color interpolator in accordance with an 
embodiment of the invention; 

[0020] Figure 6 is a flowchart detailing the color interpolation of a digital image by the 
color interpolation circuit in accordance with an embodiment of the invention; 

[0021] Figure 7 is a block diagram of a universal state machine controller in accordance 
with an embodiment of the invention; 

[0022] Figure 8A is a block diagram of a universal state machine controller in accordance 
with another embodiment of the invention; 

[0023] Figure 8B is a representative non-symmetric clock signal in accordance with an 
embodiment of the invention; 

[0024] Figure 9 is a block diagram of a programmable analog reference signal generator in 
accordance with an embodiment of the invention; 

[0025] Figure 1 OA is a flowchart detailing authentication stamping of a digital image in 
accordance with an embodiment of the invention; 

[0026] Figure 1 0B is a flowchart detailing verification of an authentication-stamped digital 
image in accordance with an embodiment of the invention; 

[0027] Figure 1 1 is a flowchart detailing the image processing of raw digital image data by 
a digital image processor in a digital camera system in accordance with an embodiment of the 
invention; 

[0028] Figure 12 is a block diagram of a digital camera system in accordance with another 
embodiment of the invention; 

[0029] Figure 13 is a block diagram of a digital camera system in accordance with yet 
another embodiment of the invention; 

[0030] Figure 14 is a block diagram of a digital camera system in accordance with another 
embodiment of the invention; and 

[0031] Figure 15 is a block diagram of a digital camera system in accordance with another 
embodiment of the invention. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0032] The present invention relates generally to digital cameras. In various aspects of the 
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invention, the digital camera includes a programmable processor capable of processing digital 
images. The processing may include digital image correction and/or digital image 
authentication stamping, a programmable source of control and timing signals and the 
capability of providing an adaptive pixel color interpolator. In another aspect of the invention, 
the processor has a system bus architecture that provides the digital camera with enhanced 
flexibility. 

[0033] In the following description, numerous specific details are set forth in order to 
provide a thorough understanding of the present invention. It will be apparent, however, to one 
skilled in the art, that the present invention may be practiced without some or all of these 
specific details. In other instances, well known structures or operations have not been described 
in detail in order to avoid unnecessarily obscuring the present invention. 

[0034] Referring initially to Figure 2A, a digital camera system 200 in accordance with one 
embodiment of the invention will be described. The digital camera system 200 includes an 
imaging unit 202 connected to a digital image processor 204 capable of producing digital 
signals representative of a captured image. The digital image processor 204 is connected to a 
local memory 206 and is capable of performing digital image processing operations upon 
received digital images. Such processing operations include, but are not limited to, image 
correction, color correction, color interpolation, as well image compression and/or image file 
formatting. The local memory 206 is capable of fast storage and retrieval of selected digital 
image files. 

[0035] In one embodiment, the digital image processor 204 is also capable of authentication 
stamping a particular digital image as well as deriving color weight factors useful in the 
operation of the imaging unit 202. A microprocessor 208 connected to the digital image 
processor 204 provides direction for the components including the digital camera system 200. 
A system memory 210 connected to the digital image processor 204 is capable of storing files 
that have passed through the image processor 204. 

[0036] The digital images upon which the digital image processor 204 operates may be 
received from any appropriate source. Typically, raw digital images (i.e., those digital images 
that have not yet had any image processing) are provided by the imaging unit 202. Since the 
system memory 210 and the local memory 206 are each capable of storing digital images, they 
too can be a source of digital images for the digital image processor 204. In this way, the 
digital camera system 200 is capable of providing the user with digital images in any desired 
stage of processing. 

[0037] The digital image processor 204 is also capable of generating a wide variety of 
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control signals and/or timing signals. Such control signals and/or timing signals are useful in 
the operation of, for example, the imaging unit 202. By providing image processing as well as 
acting as a source of control signals and/or timing signals, the digital image processor 204 
reduces the number of components required in the digital camera system 200 thereby, for 
example, commensurably reducing overall costs of manufacture of the digital camera system 
200. 

[0038] The digital camera system 200 also includes optics (not shown) such as, for example 
a lens, capable of directing light from any object to the imaging unit 202. The optics may 
include such optical systems as fiber optic transducers and fiber optic cables, refractive mirror 
assemblies, or any system or device capable of being optically coupled to the imaging unit 202. 
[0039] Figure 2B is a block diagram of a digital image processor 250 in accordance with an 
embodiment of the invention. It should be noted that the digital image processor 250 is but one 
possible embodiment of the digital image processor 204 used in the digital camera system 200 
shown in Figure 2A. Therefore, the digital image processor 250 shall be described in the 
context of the digital camera system 200 and all references included therein. 

[0040] The digital image processor 250 includes a system bus 252 capable of carrying 
various signals. Such signals include but are not limited to data signals, control signals, and 
digital image signals. In the described embodiment, the system bus 252 receives raw digital 
images from the image capture unit 202. The system bus 252 also carries control signals 
generated by a programmable control signal generator 254. In one embodiment, the control 
signals so generated are supplied to the imaging unit 202, as needed, to control stepper motors 
included therein, for example. A programmable timing signal generator 256 provides timing 
signals as needed. Such timing signals are useful in the operation of, for example, image 
sensors included in the imaging unit 202. In addition, a programmable reference signal 
generator 258 provides reference signals. Such reference signals include those used in the 
operation of, for example, any analog components included in the digital camera 200. 
[0041] The digital image processor 250 also includes first and second image processors 
260, 262 connected to the system bus 252. The image processors are structured such that they 
may provide processing in addition to or independently of one another. It should be noted that 
in alternative embodiments, the digital image processors 260, 262 may include any number of 
digital image processors as may be desired or required for a particular application. 
[0042] A local memory interface 264 connecting the system bus 252 to the local memory 
206 provides the user of the digital camera system 200 with the capability of quickly storing a 
particular digital image or series of images. Such digital images include those processed by the 
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first image processor 260 and/or the second image processor 262. In one implementation, such 
digital images include those raw digital images received from the imaging unit 202. Once 
stored in the local memory 206, the digital images are selectively available for any purpose, 
such as for example, further digital image processing by any component connected either 
directly or indirectly to the system bus 252. 

[0043] In one embodiment of the invention, the digital image processor 250 includes an 
authentication stamper 266 used to generate an authenticity stamp. Typically, the authenticity 
stamp is useful in providing photographers and/or other end-users with the capability of 
determining the authenticity of any digital image so stamped. A color interpolator 268 is 
included to provide appropriate color interpolation when full color interpolation of a digital 
image by the digital image processor 250 is desired. Alternatively, when the color interpolation 
performed by the color interpolator 268 is performed by external mechanisms connected with 
the digital image processor 250 (such as the microprocessor 208), the color interpolator 268 
may be bypassed. A microprocessor interface 270 provides an interface between the 
microprocessor 208 and the system bus 252. 

[0044] Referring next to Figure 3, a digital camera system 300 in accordance with another 
embodiment of the invention will be described. It should be noted that the digital camera 
system 300 is but one possible embodiment of the digital camera system 200 shown in Figure 
2A. The digital camera system 300 includes an image capture unit 302, an A/D converter 308 
that converts captured images to representative digital signals, and a configurable digital image 
process controller 310 arranged to perform some processing of the digital images and to provide 
various clocking and control signals to the image capture unit 302 and/or A/D converter 308. 
The digital camera system 300 further includes a local memory 31 1 for quickly storing digital 
images, a microprocessor 312, a system memory 314, a video DRAM 316 for storing video 
images, a video display controller 317, a display 318 having a touch screen 319, and an 1/0 
block 320. 

[0045] The image capture unit 302 includes an image conditioner 304 for conditioning the 
light received from the optical system before it is received at an image sensor 306. Such 
conditioning can include image focusing, image enlargement (referred to as zooming), image 
exposure, and any other suitable conditioning found useful in the formation of digital images. 
The image sensor 306 typically includes a grid, or array, of photosites. The photosites can be 
any device capable of converting incident light (in the form of photons) into useable electrical 
signals. Typically, these photosites are formed of a semiconductor material and include charge- 
coupled devices (CCD) or complementary metal oxide semiconductor (CMOS) devices. 
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Generally, each of the photosites is associated with what is referred to in the art as a pixel (short 
for picture element). The resolution of the image sensor 306 is then determined by how many 
photosites are placed upon its surface. This resolution is specified in one of two ways: by its 
dimensions in pixels or by the total number of pixels in its images. For example, the same 
digital camera may claim to be 1200 x 800 pixels, or 960-thousand pixels. 

[0046] Each of the photosites in the image sensor 306 converts light into an associated 
electrical signal. Typically, the associated electrical signal so produced is related to the 
intensity (i.e., number of photons) of the light and not necessarily the color (i.e., frequency) of 
the light that falls upon the surface of the photosite. Therefore, in order to simulate the primary 
colors of red, blue, and green, for example, an associated color filter array (CFA) (also referred 
to as a color mask) must be placed in proximity to the photosite array. One such CFA is shown 
in Table 1 for a representative 4x4 CCD array. 



Table 1 
RGRG 
GBGB 
RGRG 
GBGB 



[0047] The color filter array shown in Table 1 (known in the art as a "Beyer" pattern) 
illustrates one particular approach to forming a CFA where each photosite (and its associated 
pixel) has an associated single color filter. After exposure, the electrical signal generated by 
each photosite is converted to a digital signal taking the form of, for example, an 8-, 10-, or 12- 
bit binary number, otherwise referred to as a pixel color value. To create a 24-bit image, for 
example, interpolation is used whereby neighboring pixel color values are used to calculate the 
values for the other two primary colors for each pixel. By combining these two interpolated 
colors with the color measured by the photosite directly, the original color of every pixel is 
calculated. 

[0048] During operation of the image sensor 306, each of the photosites included in the 
image sensor 306 generates an associated analog (electrical) signal representative of a portion 
of the image rendered by the image controller 306. The analog signals are then fed to an 
analog-to-digital (A/D) converter 308. The AD converter 308 converts the received analog 
signals representative of the captured image into corresponding raw digital signals. An 
interface bus then carries corresponding raw digital signals to the digital image processor 310. 
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At some point in the processing of a raw captured image, the image is typically color 
interpolated by the digital image processor. As will be described in more detail below, one 
feature of the digital image processor illustrated in Figure 3 is that it includes a programmable 
color interpolator that may be programmed to implement a wide variety of different desired 
color interpolation schemes. 

[0049] In the embodiment shown, the digital image processor 310 is connected to the 
microprocessor 312. It should be noted that a wide variety of microprocessors may be used and 
in some implementations, it may be desirable to combine the functionalities of the digital image 
processor 310 and microprocessor 312. One useful function provided by the microprocessor 
312 referred to as data packetizing provides for efficient transmission and storage of the digital 
image data. In one implementation, the microprocessor 312 arranges the digital image data into 
discrete units such as data words formed of n data bits. In another implementation, the 
microprocessor 312 may packetize the digital image data into data packets having associated 
header portions and data portions. 

[0050] The microprocessor 312 is capable of directing the retrieval of digital images stored 
in either the system memory 3 14 or the local memory 311. Once retrieved, the microprocessor 
312 is capable of directing the sending of the digital images to any number and type of output 
devices that may be provided on the camera. In one implementation, the digital images may be 
sent by way of an 1/0 bus to the 1/0 block 320. In the described embodiment, the I/O block 320 
includes 1/0 ports such as a parallel port, a serial port, a USB port, a TV signal output port, a 
PCMCIA port, as well as a modem port. The microprocessor 312 is also capable of directing 
the video controller 317 to store in the video DRAM 316 digital video images, which are then 
sent via a display bus to a display 318 for viewing. A touch screen 319 overlaying the display 
318 is typically used to input display coordinate data to the microprocessor 312. Such 
coordinate display data is useful in coordinating user inputs resulting in improved ease of use of 
the digital camera 300. It should be noted that the microprocessor 312 is capable of parallel 
execution of these and any other instructions suitable for the operation of the digital image 
processor 310. 

[0051] The digital image processor 310 also provides image sensor control signals to the 
image sensor 306 by way of an optics control bus. Such image sensor control signals include 
but are not limited to those suitable for synchronizing the array of photosites included in the 
image sensor 306. When the array of photosites in the image sensor 306 is a CCD array, the 
CCD array requires clock signals that synchronize, for example, the reading of charge data in of 
each of the rows in the CCD array and its associated data register. These clock signals, 
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otherwise referred to as pixel clock signals, may include other pixel clock signals used to 
synchronize the CCD array. In addition to providing a programmable source of image sensor 
control signals, the digital image processor 310 provides a programmable source of reference 
signals carried by the interface bus useful in the operation of, for example, the A/D converter 
308. 

[0052] Referring next to Figure 4, one embodiment of a digital image processor 3 1 0 will be 
described. It should be noted that the digital image processor 400 is but one possible 
implementation of the digital image processors 310 shown in Figure 3. Typically, the 
integrated circuit in which the digital image processor 400 is formed is an application specific 
integrated circuit (ASIC) having various functional blocks and memory blocks included therein. 
However, in alternative embodiments it may be implemented in any suitable form including 
software and programmable logic and combinations of forms. 

[0053] The configurable image controller 400 includes a preprocessor 402, a postprocessor 
406, a local memory interface 416, a programmable universal controller 428 and a system CPU 
interface 450, all of which communicate over system bus 404. Raw image data that is received 
by the image controller 400 is first passed to the preprocessing stage 402, which typically does 
at least some preprocessing of the image data before the image data is stored in memory, 
displayed or otherwise handled. 

[0054] In the illustrated embodiment, the preprocessing stage 402 includes a uniformity 
corrector 408, a sampling filter 410, a modulation transformer 412, and a ditherer 414. The 
uniformity corrector 408 performs digital image uniformity correction using, for example, 
Photo Response Non-Uniformity (PRNU) correction and white balancing. The uniformity 
corrector 408 is arranged to correct nonuniformities in the image sensor and typically uses a 
PRNU coefficient stored in the system memory 314 to correct any sensor based non- 
uniformities in the received digital image. In the described embodiment, the uniformity 
corrector 408 is placed first since it corrects sensor errors and it is expected that almost any 
digital camera incorporating the digital image processor will want to incorporate uniformity 
correction. In the (believed to be unlikely) event that no uniformity correction is desired, then 
the scaling coefficients can be set appropriately, or alternatively, the uniformity corrector 408 
can be bypassed. 

[0055] After the uniformity correction has been applied, the received image data may be 
handled in a variety of ways under the direction of the microprocessor depending upon the 
desires of a particular camera manufacturer. One operational mode that may be desired is 
referred to herein as a "capture mode", which rapidly stores digital images in the local memory 
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3 1 1 . One appropriate capture mode may contemplate directly outputting images that have been 
processed by the uniformity corrector 408 to the local memory 311. This may be particularly 
useful when the camera it attempting to take a fast sequence of pictures. Alternatively, the 
stored digital image may be made available for immediate viewing on the display 318. 
[0056] Another operational mode that may be desired is referred to herein as a "cineview" 
mode. As will be appreciated by those familiar with digital cameras, many digital cameras do 
not use traditional optical viewfmders. Rather, images from the image sensor are presented at a 
relatively fast rate on a small display on the camera. Typically the images are color images that 
are presented at rates on the order of approximately 16 frames per second. The sampling 
(decimation) filter 410 provided to permit quick image size reductions may facilitate faster and 
more efficient viewing on smaller displays such as those found in viewfmders. By way of 
example if a captured image has dimensions of 1200 by 800 pixels, but the on-camera display is 
only 300 by 200 pixels, the amount of pixel data that needs to efficiently be transferred to the 
on-camera display for viewing is the smaller amount of data. Therefore, the sampling unit may 
be utilized to reduce the image file to the appropriate size, which speeds the processing of the 
image file and the delivery of the image data to the display. The sampling filter 410 receives 
input data from the uniformity corrector 408. The sampling filter 410 is programmable so that 
the amount of decimation may be either set by the manufacturer or programmed by the 
microprocessor. This permits the same sampling filter to be used with a wide variety of digital 
cameras, which may have very different image sensing pixel arrays and very different on- 
camera displays. The sampling filter may also be used to permit pictures having different 
image resolutions taken by a camera to be displayed on a single display. 

[0057] While still in the cineview mode, the output of the sampling filter 410 is output to 
the modulation transformer 412. The modulation transformer 412 is connected to a ditherer 414 
and is capable of correcting image degradation caused by the sampling filter 410 The 
modulation transformer 412 outputs the corrected digital image to the ditherer 414 which is 
arranged to perform anti-aliasing suitable to provide for better viewing on, for example, an 
LCD d 1S play. The ditherer 414 outputs the resulting image to the system bus 404 where it is 
available for display or even potentially, the local memory interface 416 for storing in the local 
memory 311. The pipelined architecture of the preprocessor permits the lower resolution 
images to be quickly processed, which is particularly desirable in the cineview mode so that the 
Splayed images better simulate what a user might see through a conventional viewfinder. 
[0058] The postprocessor 406 includes a number of processing blocks that implement 
specific transformations and other processing of an image that a camera manufacturer may 
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desire to provide with the camera. In the embodiment shown, the postprocessor 406 includes a 
color interpolator 422, an RGB reconstruction block 424, a digital compressor 426 and a color 
pattern data buffer 427. The postprocessor is particularly useful in processing the digital 
images for printing or display. 

[0059] The color interpolator 422 provides color correction to the captured image. 
Specifically, as described above with reference to Table 1, each pixel of the raw captured 
images typically indicates the intensity of the incoming light at one specific primary color as 
determined by the color filter array chosen by the manufacturer. The interpolator 422 is then 
used to estimate the values of the other two colors for each pixel. To do this, color correction 
factors suitable for estimating the values of the other colors for each pixel are determined. The 
actual values of the color correction factors chosen are typically based on a variety of factors 
including the color filter array used, the type of interpolation desired, and the designers sense of 
optimal color balance. In one aspect of the invention, the color interpolator 422 is capable of 
using color correction factors derived by, for example, firmware in the microprocessor 312, 
associated with a CFA included in the image sensor 306. This feature and the structure of one 
embodiment of the color interpolator 406 will be described in more detail below with reference 
to Figure 5. By providing the capability of deriving color correction factors for any CFA, the 
configurable image process controller 310 may be used by any digital camera manufacturer, 
regardless of their specific image sensor, color interpolation scheme, etc., in a digital camera 
system. In this way, the configurable image process controller 310 will significantly reduce 
development time and costs, as well as component costs since specific ASICs are no longer 
necessary. 

[0060] When not RGB based, the color corrected digital images are first output to an RGB 
reconstruction block 424 and then passed to a digital compressor 426. Otherwise, the color 
corrected digital images are sent directly to the digital compressor 426. Such digital 
compression techniques include those techniques based upon color space conversion, such as, 
for example, JPEG. Once digitally compressed, the compressed image files are then passed to 
the system bus 404 where, in one implementation, they may be stored in the system memory 
314 and/or the local memory 31 1. A color pattern data buffer 427 connected to the system bus 
404 capable of storing appropriate color interpolation input data is operatively connected to the 
color interpolator 422. Such color interpolation input data may include the number of pixels in 
the image sensor array, the particular CFA used with the image sensor array, as well as any 
particular image filtering and other appropriate digital image filtering values. 
[0061] The described architectures provides camera manufactures with a great deal of 
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flexibility in directing the data flow within the camera, as well as in defining the camera's 
functionalities and designs. For example, a raw digital image processed by the uniformity 
corrector 408 may be directly stored in the local memory 311 and later retrieved for further 
processing. Alternatively, the image may be passed through the sampling filter 410 and on 
through the preprocessor 402 prior to either storage in the local memory 3 1 1 or being passed for 
direct viewing on a viewfinder, for example. In this case, it may be desirable to provide a data 
buffer (not shown) to hold the image before the sampling filter 410. 

[0062] Images that are stored in either memory may be retrieved and processed as desired. 
In some cases, the processing may be direct, while in other cases, the processing may be staged. 
For example, a PRNU and white balance processed image stored in memory may be retrieved 
and processed by the remainder of the preprocessor 402, including the sampling filter 410, the 
modulation transfer function block 412, as well as the dithering block 414. In other situations, a 
stored image may be retrieved and passed to the postprocessor 406, authentication stamper 418 
or any of the other processing blocks that has direct or indirect access to the system bus. 
[0063] In still another operational mode, the postprocessor 406 may receive digital image 
files directly from the preprocessor 402 by way of the system bus 404. It should be noted that 
the preprocessor 402 and the postprocessor 406 may concurrently process digital image files 
associated with different captured images. 

[0064] In the described embodiment, the system memory 314 includes a system dynamic 
random access memory (DRAM), a system read only memory (ROM), random access memory 
(RAM), as well as any other appropriate volatile or nonvolatile storage media. Such storage 
media includes but is not limited to memory cards such as, for example, "floppy" disks, flash 
EPROM cards, R/W compact disc (CD), SmartMedia™ and the like. 

[0065] It should be noted that due to the efficient architecture of the digital image processor 
300, all operations are parallel in nature in that all may be performed substantially 
simultaneously. 

[0066] As pointed out above, the primary purpose of the postprocessor 406 is to prepare the 
digital images for printing or display on an external device. In many circumstances, the 
camera's user may not have ready access to high quality printers suitable for printing 
photographs. Thus, it may be desirable for the user to send an electronic copy of a particular 
captured image to a commercial entity that prints the photograph. The postprocessor 406 can 
do the postprocessing necessary for such third party printing. However, it may be more 
efficiently and perfectly done by the third party which may have more sophisticated processing 
abilities, such as the use of a more sophisticated color interpolator. Thus, the described camera 
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also supports another operational mode referred to herein as an off-line processing mode. In 
this mode, digital images stored in the local memory 311 or the system memory 314 can be 
output to any .appropriate I/O port included in the I/O block 320. In this way, additional digital 
image processing available by external devices may be used to complete the digital image 
processing. Such digital image processing may include color correction, RGB reconstruction 
(if necessary), MTF, dithering, etc. In this way, the user is able to take advantage of digital 
image processing capabilities beyond those available using the digital camera 300. In addition, 
by digitally compressing the digital image(s) before being transmitted over, for example, the 
Internet, valuable time and resources are conserved. 

[0067] Still referring to Figure 4, the programmable universal controller 428 connected to 
the system bus 404 is capable of selectively generating control and reference signals. Such 
control signals include but are not limited to those used to in the operation of stepper motors, 
for example, included in the image conditioner 304. Other signals include periodic signals 
(such as clock signals, both symmetric and non-symmetric) used in the operation of the image 
sensor 306. When the image sensor 306 includes a CCD array, these periodic signals include 
clock signals referred to as pixel clock signals useful in the operation of the CCD array. It 
should be noted that due to the programmable nature of the programmable universal controller 
428, a wide variety of image sensors can be accommodated by the digital image processor 310 
without the need to resort to expensive and time consuming fitting procedures. 
[0068] The programmable universal controller 428 is also capable of generating reference 
signals useful in the operation of analog components included in the digital camera 300. Such 
analog components for which the analog reference signals may be used include the A/D 
converter 308. By way of example, in one embodiment, the image capture unit 302 includes an 
automatic rangefinder useful in determining the distance to the object being photographed. 
Typically, the rangefinder generates signals related to the measured distance that are sent to an 
evaluator that determines whether or not the object is within the proper range. If not, 
microstepper controller signals are fed back to appropriate motors that control image' 
conditioners, such as focus and zoom. More particularly, in one example, based upon the 
evaluation, a focus signal generator 430 and a zoom signal generator 434 generate appropriate 
micro-stepper control signals. These micro-stepper control signals are then fed to the image 
conditioner 304 by way of the optics control bus. In this way, the programmable universal 
controller 428 provides for rapid and automatic focus and zoom control without substantially 
increasing the number of components within the digital camera 300. Additionally, the digital 
image processor 310 is capable of reading the digital image from the image sensor 306 and 
analyzing a central portion of the array of photosites for proper focus. 
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[0069] In one embodiment of the invention, a data buffer 432 associated with the focus 
signal generator 430 takes the form of a look up table (LUT) having stored micro-stepper signal 
values corresponding to the received distance signals. A data buffer 436 associated with the 
zoom signal generator 434 can also be a LUT having stored micro-stepper signal values 
associating the received distance signals to the proper zoom value. In addition to generating 
required micro-stepper controller signals, the programmable universal controller 428 supplies 
periodic signals, such as for example, timing signals. Such timing signals may be generated in 
response to signals received from, for example, an f-stop controller, a light meter, a shutter 
controller as well as an associated flash controller included in, for example, the image 
conditioner 304. 

[0070J By way of example, the image capture unit 302 may include a photometer 
responsive to the level of ambient light. The photometer may have an ambient light level 
threshold below which signals are sent to the programmable universal controller 428 indicating 
that the light level is insufficient to produce an image of the desired quality. At this point, a 
shutter timing generator 438 and an f-stop timing generator 440, for example, generate 
appropriate timing signals using a clock and associated timing data. 

[0071] In one embodiment of the invention, the shutter timing generator 438 and the f-stop 
timing generator 440 are associated with a shutter timing data buffer 442 and an f-stop timing 
data buffer 444, respectively. The shutter timing data buffer 442 and the f-stop timing data 
buffer 444 are each capable of storing any data appropriate to the generation of the respective 
timing signals. A clock circuit 452 connected to the system bus 404 is used by the shutter 
timing generator 438 and the f-stop timing generator 444 to generate the required control 
signals. In this way, the digital image processor 400 provides an integrated automatic approach 
to the operation of the digital camera system 300. 

[0072] In addition to providing timing and micro-stepper control signals, the programmable 
universal controller 428 includes a universal state machine controller 446 capable of providing 
both symmetric and non-symmetric periodic signals. Such periodic signals may include clock 
signals such as those pixel clock signals used to synchronize the operation of, for example, an 
exemplary CCD array included in the image sensor 306. Such pixel clock signals include 
integration clocks, reset clocks, shift clocks, and any other periodic signals deemed appropriate 
by, for example, an exemplary CCD array manufacturer. 

[0073] The programmable universal controller 428 also includes a universal analog 
reference signal generator 448 capable of generating any required analog reference signals. 
These analog reference signals may be used in, for example, the operation of the A/D converter 
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308. Both the universal analog reference signal generator 448 and the universal state machine 
controller 446 have associated input data registers operatively connected to the system bus 404. 
[0074] In one embodiment of the invention, the universal analog reference signal generator 
448 and the universal state machine controller 446 each have a set of data registers 447 and 
449, respectively, for storing data appropriate to the operational mode of the digital camera 300. 
By way of example, in what is referred to herein as the cineview mode, a stream of digital 
images are processed at a rate sufficient to simulate motion (typically in the range of 
approximately 10 fps to approximately 20 fps). In order to provide proper control and timing 
signals, data suitable for operating the digital camera 300 in cineview mode are stored in 
associated ones of the data registers 447 and the data registers 449. 

[0075] The system CPU interface 450 connects the system bus 404 to the microprocessor 
312 and provides access to all internal registers and data buffers included in the digital image 
processor 400. In this way, the microprocessor 312 may set all internal registers and/or data 
buffers as may be required for proper operation. 

[0076] Figure 5 is a functional block diagram of a color interpolation circuit 500 in 
accordance with an embodiment of the invention. It should be noted that the color interpolation 
circuit 500 is but one possible embodiment of the color interpolator 422 used in the digital 
image processor 400 shown in Figure 4. In the described embodiment, the color interpolation 
circuit 500 is capable of receiving a variety of color interpolation input data and determining the 
weights that are to be used in processing raw input pixel signals into re-sampled pixel signals 
that have full color at each location. The color interpolation input data may include the number 
of photosites (or pixels) on the active surface of the image sensor array, the associated color 
filter array (CFA), as well as any desired filtering of the re-sampled resulting image. 
[0077] The color interpolation circuit 500 includes a pixel color weight generator 502 
capable of generating the pixel color weights that, taken together, form pixel color weight 
matrices used to generate the re-sampled image. The pixel color weight generator 502 receives 
the color interpolation input data, such as the number of pixels in the image sensor array, the 
associated CFA of the particular image sensor array, as well as any desired filtering. The pixel 
color weight generator 502 then uses the received color interpolation input data to form a 
multiplexed array of pixel color weights. The multiplexed array of pixel color weights are then 
stored in a pixel color weight matrix buffer 504 which forms an input to a configurable 
convolver 506. The configurable convolver 506 in turn receives the raw image data and 
operates to form the re-sampled resulting image having full color at every pixel. 
[0078] Figure 6 is a flowchart detailing the color interpolation 600 of a digital image by the 
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color interpolation circuit 500 in accordance with an embodiment of the invention. It should be 
noted that the flowchart illustrates but one possible process implemented in the color 
interpolation circuit 500 used in the digital image processor 400 shown in Figure 5. The color 
interpolation of the digital image begins with the pixel weight generator 502 receiving 
appropriate image sensor data 610. The image sensor data may include the number of pixels in 
the image sensor array as well as the particular color filter array associated with the image 
sensor array. The input data may also include optional digital image filtering selected by the 
user. Once received, a determination is made if new pixel color weight factors are required 
620. If it is determined that new pixel color weight factors are required, they are generated 630, 
in one embodiment of the invention, by firmware included in the microprocessor 312 using the 
image sensor input data. These pixel color weight factors may be generated by any number of 
techniques well known to those skilled in the art. One such technique referred to as bi-linear 
interpolation uses known pixel colors to derive the unknown pixel colors. Another well known 
technique referred to as bi-cubic interpolation may be used when higher color fidelity is 
required. Once the pixel color weight factors have been derived using any suitable technique, 
they are stored 640 in a pixel color weight buffer where they remain until needed. A 
configurable convolver then uses the stored pixel color weight factors to process 650 received 
digital images and generate full color digital images. 

[0079] Figure 7 is a block diagram of a universal state machine controller 700 in 
accordance with an embodiment of the invention. It should be noted that the universal state 
machine controller 700 is but one possible embodiment of the universal state machine controller 
446 used in the digital image processor 400 shown in Figure 4. The universal state machine 
controller 700 is capable of providing both symmetric and non-symmetric clock signals as well 
as pulse signals. By symmetric clock signals it is meant those clock signals having symmetric 
waveforms whereas non-symmetric clock signals have non-symmetric waveforms. Such clock 
signals may include those pixel clocks used to synchronize the operation of the CCD array 
included in the image sensor 306. 

[0080] The universal state machine controller 700 includes a fast clock 702 capable of 
generating precise symmetric waveforms. Typically, the fast clock period is approximately 10 
ns but may range as low as approximately 1 ns, or as may be required by the particular CCD 
array for which the clock signals are being generated. In the embodiment shown in Figure 7, 
the fast clock 702 forms an input to a logic unit 704 having input lines 706 through 712 capable' 
of carrying input signals derived from data stored in the data registers 447. The logic unit 702 
uses such input signals to form the desired symmetric or non-symmetric clock signal. 
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[0081] To form a symmetric clock signal, one implementation of the universal state 
machine controller 700 provides for the input line 706 to supply an initial state signal indicative 
of a high going clock signal or a low going clock signal. The input line 508 supplies a period 
signal indicative of the number of ticks (each tick being equivalent to a single period of the fast 
clock 702) for which the desired clock signal generated by the universal state machine 
controller 700 extends. The input line 710 supplies a first change signal indicative of the tick at 
which the polarity of the clock signal generated by the universal state machine controller 700 
changes (the half cycle of the clock signal). 

[0082] If, however, a non-symmetric clock signal is desired, the universal state machine 
controller 700 provides for the input line 712 to carry a second change signal indicative of the 
tick at which the clock signal changes polarity in reference to the first change signal. 
[0083] It should be noted that the logic unit 704 may take the form of any programmable 
circuit. The programmable nature of the logic block 704 provides the universal state machine 
controller 700 with a wide range of operable modes. 

[0084] Figure 8A is a block diagram of a universal state machine controller 800 in 
accordance with another embodiment of the invention. The universal state machine controller 
800 includes a counter buffer 802 connected to a fast clock 804 capable of acting as a counter. 
The counter buffer 802 is also operably connected to a memory unit 806 capable of storing a 
representation of a desired clock signal waveform. 

[0085] In operation, a representation of the desired clock signal is stored in the memory unit 
806. One such representation shown in Figure 8A provides the universal state machine 
controller 800 with data sufficient to form the non-symmetric clock signal shown in Figure 8B. 
The universal state machine controller 800 forms the non-symmetric clock signal by applying 
the stored representation to the counter buffer 802. The counter buffer 802 in turn acts in 
conjunction with the fast clock 804 to form the associated non-symmetric clock signal. It 
should be noted that in this particular example, a logical "1" stored in the memory 806 
corresponds to logical CLOCK HIGH while a logical "0" corresponds to a logical CLOCK 
LOW. 

[0086] It should also be noted that the counter buffer 802 and the memory 806 can be any 
form of memory capable of storing data consistent with the operation of the universal state 
machine controller 800. Such memories can include but are not limited to static random access 
memories, dynamic random access memories, and any other suitable volatile or non-volatile 
memory device. 

[0087] Figure 9 is a block diagram of a programmable analog reference signal generator 
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900 in accordance with an embodiment of the invention. It should be noted that the 
programmable analog reference signal generator 900 is but one possible embodiment of the 
programmable analog reference signal generator 448 used in the digital image processor 400 
shown in Figure 4. The programmable universal controller 900 includes a universal analog 
reference signal generator 902 coupled to a programmable data buffer 904 capable of 
generating analog reference signals. It should be noted that the programmable data buffer 904 
is but one possible embodiment of the register set 449 shown in Figure 4. 

[0088] In one embodiment of the invention, the universal analog reference signal generator 
902 takes the form of a pulse width modulation (PWM) block represented as PWM 906. The 
PWM 906 uses data stored in the programmable data buffer 904 in conjunction with a fast clock 
908 to generate any desired analog reference signal. The analog reference signals generated can 
be selectively formed to meet the requirements of the particular operating mode of the digital 
camera 300. In the case where the digital camera 300 is operating in the capture mode, the 
programmable data buffer 904 can supply data appropriate to that mode of operation. In 
another case where the digital camera 300 is operating in, for example, the cineview mode, the 
programmable data buffer 904 can supply appropriate data accordingly. 

[0089] Figure 10A is a flowchart detailing authentication stamping of a digital image in 
accordance with an embodiment of the invention. It should be noted that the flowchart 
illustrates but one possible process implemented in the authentication stamper 418 used in the 
digital image processor 400 shown in Figure 4. Therefore, the authentication stamping shall be 
described in context of the digital image processor 400 and all references included therein. 
First, a digital image to be authentication stamped along with associated useful authenticating 
information (sometimes referred to as private camera information) are obtained 1005. The 
digital image to be authentication stamped and the authenticating information are then 
processed 1010 using for example, a one-way HASH algorithm. The resulting image digest 
1015 is encrypted 1020 using a secure key to form an digital authentication stamp 1025 which 
is appended to the digital image 1030. 

[0090] Figure 10B is a flowchart detailing verification of an authentication stamped digital 
image in accordance with an embodiment of the invention. First, a digital image having an 
associated authenticity stamp is received 1050. Next, decrypting the associated authenticity 
stamp using a public key 1055 forms a first image digest. A second image digest is also formed 
by processing the digital image to be verified, using for example a one-way HASH algorithm 
1060. Next, the first image digest and the second image digest are compared 1065. Finally, 
verification of the digital image based upon the comparing of the image digests is performed 
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1070. In one implementation of the invention, if the image digests are equal, then the digital 
image is verified. Alternatively, if the image digests are not equal then the digital image is not 
verified. 

[0091] The authenticity stamp is useful for many purposes including, for example, 
authenticating the source camera, image author, and image date of any digital image so 
stamped. It should be noted that the secure key is associated with only the digital camera 
system 300 into which the digital image processor 400 is installed. Any other digital camera 
system into which the digital image processor 400 is installed will have a different secure key. 
In this way, any digital image produced by a particular digital camera system may be uniquely 
ascribed to only that particular digital camera system. In other implementations, the 
authentication stamper 418 may include user specific information such as, for example, user 
name, user address, camera serial number, manufacturing date and/or code that may be used to 
further identify the source of the digital image of interest. 

[0092] Figure 1 1 is a flowchart detailing the image processing I 100 of raw digital image 
data by a digital image processor 400 in a digital camera system 300 in accordance with an 
embodiment of the invention. The image processing of raw digital image data begins with an 
image sensor within the digital camera generating a raw analog digital data that a converter then 
converts to raw digital image data. A preprocessor receives the raw digital image data 1105. 
The first processing stage then corrects 1 1 10 any non-uniformities in the digital image. Such 
non-uniformities correction may be accomplished using PRNU and white balance techniques 
where PRNU coefficients are stored in memory. Next, if the digital camera is operating in 
capture mode 1115, the corrected digital image is stored in memory 1120. If, however, the 
digital camera is not operating in capture mode 1115, then it is determined whether the 
corrected digital image is to be authentication stamped 1 125. If the corrected digital image is to 
be authentication stamped, then the corrected digital image is forwarded to the authentication 
stamper 1130. If it is determined that the image is not to be authentication stamped 1125, then 
the corrected digital image is fully preprocessed by the preprocessor 1135. Next, it is 
determined if the digital image is to be post processed on-chip 1 140. By on-chip it is meant that 
post processing is performed by the digital image processor. If it is determined that the post 
processing is performed off-chip, the digital images are sent to an off chip processor 1145. If it 
is determined that the post processing is to be done on-chip 1 140, it is then determined if new 
pixel color weight factors are required 1 150. If it is determined that new pixel color weight 
factors are required, then appropriate image sensor data is received 1 155 and used to derive new 
pixel weight factors 1 160 associated with the image sensor. In either case, the digital image is 
then color corrected 1165, and the color corrected digital image is output to the system bus 
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[0093] Figure 12 is a block diagram of a digital camera system 1200 in accordance with 
another embodiment of the invention. The digital camera system 1200 is formed of the digital 
camera system 300 shown in Figure 3 wherein the microprocessor 312 has been replaced by 
specialized processors. Such specialized processors include a video processor 1202 capable of 
processing viewable images as well as a system processor 1204 capable of directing the 
operations of the digital camera system 1200. By providing specialized processors, the digital 
camera system 1200 is well suited for use in more specialized applications where particular 
processing needs are important. An application requiring high speed video processing 
unavailable with more general application processors would be well served by the digital 
camera system 1200. 

[0094] Figure 1 3 is a block diagram of a digital camera system 1 300 in accordance with yet 
another embodiment of the invention. The digital camera system 1300 is formed of the digital 
camera system 300 wherein the microprocessor 312 and the digital image processor 310 have 
been combined into a microprocessor 1302. In this arrangement, the digital camera system 
1300 has a higher degree of integration providing for fewer components in the manufacture of 
the digital camera system 1300. 

[0095] It should also be noted that the digital image processor is capable of supporting any 
suitable number imaging arrays included in an associated image sensor. 

[0096] Figure 14 is a block diagram of a digital image processor 1400 in accordance with 
an embodiment of the invention. The digital image processor 1400 is capable of processing raw 
digital images from an image sensor that includes three imaging arrays where each imaging 
array is arranged to respond to, for example, a single primary color. As shown in Figure 14, the 
digital image processor 1400 is a straightforward modification of the digital image processor 
400 shown in Figure 4 and the digital camera 300 shown in Figure 3. The digital image 
processor 1400 receives the raw digital images from three image sensors and simultaneously 
processes the raw digital images. Three separate preprocessors 402-1, 402-2, and 402-3 receive 
separate raw digital images from associated image sensors and process the respective raw 
digital images accordingly. The processed images are then combined using, for example, 
firmware included in the microprocessor 312 before being stored in the local memory 311 
and/or the system memory 314. In this way, a camera manufacturer is capable of producing a 
digital camera capable of producing very precise color photography without resorting to 
expensive and time consuming fitting procedures. It should be noted that the authentication 
stamper 41 8 has been omitted for sake of clarity only, and an authentication stamper may in fact 
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be used to authentication stamp any image the digital image processor 1400 processes. 
[0097] Figure 1 5 is a block diagram of a digital image processor 1 500 capable of processing 
digital images from three image sensors in accordance with another embodiment of the 
invention. The digital image processor 1500 is capable of processing the raw digital images 
from three image processors using a multiplexer 1502 connected to three data buffers 1504-1 
through 1504-3 capable of staging raw digital images from the three image sensors included in 
the digital camera. The three raw digital images staged in the data buffers 1504-1, 1504-2, and 
1504-3 are received at the multiplexer 1502 based upon a selection signal generated by a 
selector 1506. Once selected, the raw digital image is passed to the preprocessor 402 for 
suitable processing. 

[0098] The invention has numerous advantages. One advantage of the invention is that the 
digital image processor can be used by any digital camera manufacturer, regardless of their 
specific image sensor, color interpolation scheme, etc., in a digital camera. The digital image 
processor will also significantly reduce development time and costs, as well as component costs 
since specific ASICs no longer must be provided. 

[0099] Another advantage of the invention is that the system bus architecture provides for 
flexible operation of the digital camera. The system bus architecture also provides the digital 
camera manufacturer the capability of economically configuring the digital camera as desired. 
In this way, the camera manufacturer is able to offer a wide variety of digital camera 
configurations without the need to resort to expensive and time consuming fitting procedures. 
[00100] Yet another advantage of the invention is that it provides the digital camera user 
with on demand specialized imaging modes such as the capture mode and the cineview mode. 
Providing these specialized imaging modes makes the taking of quality photos using the digital 
camera much easier. In addition, these and other specialized imaging modes make the use of 
the digital camera more cost and time effective since the user is less likely to waste time and 
effort on photos that will not be used. 

[0100] Although the foregoing invention has been described in some detail for purposes of 
clarity of understanding, it will be apparent that certain changes and modifications may be 
practiced within the scope of the appended claims. It should be noted that there are many 
alternative ways of implementing the present invention. For example, the system bus has been 
described as carrying data signals, control signals, and digital image signals. In alternative 
embodiments, a separate control or other bus could be provided to carry some of these signals. 
It is therefore intended that the following appended claims be interpreted as including all such 
alterations, permutations, and equivalents as fall within the spirit and scope of the present 
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invention. 



